-
Notifications
You must be signed in to change notification settings - Fork 28.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-26219][CORE] Executor summary should get updated for failure jobs in the history server UI #23181
Conversation
Test build #99466 has finished for PR 23181 at commit
|
Jenkins, retest this please |
cc @vanzin Kindly review |
Test build #99468 has finished for PR 23181 at commit
|
Jenkins, retest this please |
Test build #99482 has finished for PR 23181 at commit
|
Jenkins, retest this please |
Test build #99492 has finished for PR 23181 at commit
|
val stage = new StageInfo(1, 0, "stage", 4, Nil, Nil, "details") | ||
listener.onJobStart(SparkListenerJobStart(1, time, Seq(stage), null)) | ||
listener.onStageSubmitted(SparkListenerStageSubmitted(stage, new Properties())) | ||
isLiveSeq.foreach { live: Boolean => |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
When doing things like this I prefer to invert the logic.
Seq(true, false).foreach { live =>
test(s"blah blah blah (live = $live)") {
}
}
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks. Updated.
listener.onStageSubmitted(SparkListenerStageSubmitted(stage, new Properties())) | ||
isLiveSeq.foreach { live: Boolean => | ||
val testConf = if (live) { | ||
conf.clone.set(LIVE_ENTITY_UPDATE_PERIOD, Long.MaxValue) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: clone()
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done
Jenkins, retest this please |
Test build #99524 has finished for PR 23181 at commit
|
Test build #99523 has finished for PR 23181 at commit
|
Merging to master / 2.4. |
This didn't merge cleanly to 2.4, please open a PR against that branch if you want it there. |
Thanks @vanzin. I will open a PR in 2.4 branch |
…obs in the history server UI The root cause of the problem is, whenever the taskEnd event comes after stageCompleted event, execSummary is updating only for live UI. we need to update for history UI too. To see the previous discussion, refer: PR for apache#23038, https://issues.apache.org/jira/browse/SPARK-26100. Added UT. Manually verified Test step to reproduce: ``` bin/spark-shell --master yarn --conf spark.executor.instances=3 sc.parallelize(1 to 10000, 10).map{ x => throw new RuntimeException("Bad executor")}.collect() ``` Open Executors page from the History UI Before patch: ![screenshot from 2018-11-29 22-13-34](https://user-images.githubusercontent.com/23054875/49246338-a21ead00-f43a-11e8-8214-f1020420be52.png) After patch: ![screenshot from 2018-11-30 00-54-49](https://user-images.githubusercontent.com/23054875/49246353-aa76e800-f43a-11e8-98ef-7faecaa7a50e.png) Closes apache#23181 from shahidki31/executorUpdate. Authored-by: Shahid <shahidki31@gmail.com> Signed-off-by: Marcelo Vanzin <vanzin@cloudera.com>
…or failure jobs in the history server UI Back port the commit apache#23181 into Spark2.4 branch Added UT Closes apache#23191 from shahidki31/branch-2.4. Authored-by: Shahid <shahidki31@gmail.com> Signed-off-by: Marcelo Vanzin <vanzin@cloudera.com>
…or failure jobs in the history server UI Back port the commit apache#23181 into Spark2.4 branch Added UT Closes apache#23191 from shahidki31/branch-2.4. Authored-by: Shahid <shahidki31@gmail.com> Signed-off-by: Marcelo Vanzin <vanzin@cloudera.com>
…or failure jobs in the history server UI Back port the commit apache/spark#23181 into Spark2.4 branch Added UT Closes #23191 from shahidki31/branch-2.4. Authored-by: Shahid <shahidki31@gmail.com> Signed-off-by: Marcelo Vanzin <vanzin@cloudera.com> (cherry picked from commit 90fcd12)
What changes were proposed in this pull request?
The root cause of the problem is, whenever the taskEnd event comes after stageCompleted event, execSummary is updating only for live UI. we need to update for history UI too.
To see the previous discussion, refer: PR for #23038, https://issues.apache.org/jira/browse/SPARK-26100.
How was this patch tested?
Added UT. Manually verified
Test step to reproduce:
Open Executors page from the History UI
Before patch:
After patch: